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METHOD AND SYSTEM FOR CLASSIFYING 
IMAGE ELEMENTS 

Reference to Related Documents 
[0001 ] This application claims benefit of priority under 35 U.S.C. §1 1 9(e) to 
U.S. provisional application serial No. 60/265,544, filed on February 1, 2001 , 
the entire contents of which are incorporated herein by reference. 

Technical Field 

[0002] The present invention relates generally to image processing and, 
more particularly, to methods, systems, and machine-readable media for 
classifying image elements. 

Background of the Invention 
[0003] ' Many images are produced using computerized methods that do 
not rely on a pixel-based representation of an image. Text processing software, 
for instance, represents an image using structured page information that 
describes high-level elements of the image, such as text, fonts, colors, 
embedded images, etc. This structured page information comes in a large 
variety of file formats such as MSWord™ doc, Adobe™ PDF, or PostScript™ 
files. When printed or otherwise rendered, the information may be converted 
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into a sequence of overlaid image elements that incrementally construct the 
image. 

[0004] There is often a need to compress, i.e., encode, these images. 
Generally, the image elements are first classified as either foreground or 
background based on some classification criteria. After classification, the 
foreground is encoded at a higher resolution because it contains the elements 
of interest. The background, on the other hand, is typically encoded at a lower 
resolution since it contains elements of less interest. Such a coding strategy is 
well known in the art, e.g. in MPEG, JPEG, etc. Thus, the quality of the 
element classification greatly affects the compression ratio and video quality of 
these images. As such, it is important to perform the classification effectively. 

[0005] Current element classification approaches for images rendered from 
structured page information include classifying all the text in the image as the 
foreground and all other details as the background, classifying all the 
monochrome elements as the foreground and all others as the background, and 
classifying the first element drawn as the background and all others as the 
foreground. However, all of these approaches are ineffective, particularly for 
geographical maps, because the elements of interest are sometimes rendered 
such that they meet the criteria for background classification when, in fact, they 
are foreground elements. As a result, these elements of interest are 
erroneously encoded at a lower resolution. As such, the compression efficiency 
and video quality of these elements significantly drop. 

[0006] Accordingly, there is a need in the art for an effective way to classify 
image elements, in general, and image elements rendered from structured page 
information, e.g., electronic documents, in particular. 
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Summary of the Invention 
[0007] The present invention provides a method for classifying an image 
element as one of a plurality of categories. The method includes classifying the 
image element based on a ratio between an unoccluded perimeter of the image 
element and a perimeter of the image element having been occluded by other 
image elements. The image element thereafter may be coded according to a 
coding scheme associated with the category to which the image element has 
been classified. 

Brief Description of the Drawings 
[0008] Figure 1 A is a block diagram of a system embodying the present 
invention; 

[0009] Figure 1 B is a flowchart of an embodiment of a method of the 
present invention; 

[001 0] Figure 2 is a flowchart of exemplary steps for calculating an image 
element perimeter; 

[001 1] Figure 3 illustrates an exemplary perimeter calculated according to 
the method of Figure 2; 

[001 2] Figure 4 is a flowchart of exemplary steps for calculating an 
occluded and unoccluded perimeter; 

[0013] Figures 5A-5C illustrate exemplary occluded and unoccluded 
perimeters calculated according to the method of Figure 4; 

[001 4] Figure 6 illustrates the unoccluded perimeter calculation; 
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[001 5] Figures 7A and 7B are flowcharts of exemplary steps for calculating 
a color difference of an image element; 

[001 6] Figure 8 is a flowchart of another embodiment of a method of the 
present invention; 

[001 7] Figure 9 illustrates an exemplary electronically produced image 
decomposed as a sequence of elementary image elements drawn on top of 
each other, upon which the methods of Figures 1B or 8 may be applied; and 

[001 8] Figure 1 0 is a block diagram of one embodiment of a computer 
system that can implement the present invention. 

Detailed Description 
[001 9] Embodiments of the present invention provide a technique for 
classifying image elements as one of a plurality of categories based on a ratio 
between an unoccluded perimeter of the element and an occluded perimeter of 
the element. The occluded perimeter of an image element typically differs from 
the unoccluded perimeter of the element when other image elements are 
overlaid upon it. Thus, the ratio can provide a good estimate of an amount to 
which an image element is blocked by other image elements. "Pushing" coding 
errors to highly occluded image elements can effectively improve perceived 
quality of a highly compressed image. 

[0020] The present invention may be applied to image compression, where 
image elements may be classified as belonging to one of foreground and 
background layers. Figure 1 A is a block diagram of an image compression 
system embodying the present invention. The system includes a rendering 
engine 10, a classifier 20, a foreground coder 30, and a background coder 40. 
The rendering engine 10 generates the image to be compressed. The 
rendering engine 10 can be, but is not limited to, a print driver or text processing 
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software. The classifier 20 processes the image by classifying its image 
elements as either foreground or background elements. The elements 
classified as foreground may be coded according to any coding technique by 
the foreground coder 30. Similarly, the elements classified as background may 
be coded according to any coding technique by the background coder 40. 
Exemplary coding techniques are well known in the art. Such techniques may 
be used independently or in conjunction with coding techniques described in 
"High Quality Document Image Compression with DjVu", by L. Bottou, et.al., 
Journal of Electronic Imaging. 7(3): 410-425, 1998, for example. Typically, the 
foreground and background coding techniques will be selected to provide 
compression of the foreground image elements with low loss of image quality. 
Whereas, the background image elements will be coded with higher 
compression and relatively higher loss of image quality. 

[0021] It is to be understood that the image compression application is for 
exemplary purposes only, as the present invention may also be used in any 
application where image element classification is performed. It is to be further 
understood that the number of categories is not limited to two. 

[0022] In exemplary image compression, a low ratio between unoccluded 
and occluded perimeters of an image element may indicate that if the element 
were classified as belonging to the background layer, any coding errors with 
respect to the element's boundaries are likely to be obscured by other occluding 
elements from the foreground layer. Thus, the coding errors are unlikely to be 
noticed. By contrast, a high ratio between unoccluded and occluded perimeters 
of the image element may indicate that coding errors with respect to the 
boundaries are likely to be observed because they will not be obscured by other 
elements. In this event, it may be appropriate to code the element in the 
foreground layer. Thus, the present invention provides a higher compression 
ratio and improved visual quality of compressed images. 
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[0023] The number of bits for encoding foreground elements may be 
proportional to the perimeter Preluded of the visible part of the element (i.e., after 
removing the element shape portions that are occluded by other elements). 
The edges of background elements that arise from occlusions by foreground 
elements may be defined by the boundary of the occluding foreground 
elements. So, the number of bits for encoding background elements may be 
reduced by excluding the occluded parts of the background elements, which 
may be already encoded as part of the foreground elements. Thus, the number 
of bits for encoding background elements may be proportional to the length 
Punocciuded of the perimeter segments that do not result from occlusions by 
foreground elements. Furthermore, the proportionality coefficient depends on 
the color differences along the element boundary. 

[0024] Figure 1 B is a flowchart of a method according to an embodiment of 
the present invention. According to the invention, an image classification 
system calculates an unoccluded perimeter of an image element, Precluded, 
(100) and a perimeter of the image element when occluded by other image 
elements, Preluded (110). The system compares a ratio of these calculated 
values to a predetermined threshold 7(120): 

p 

unoccluded ^ y 

D \'l 

'occluded 

If the ratio of Punocciuded to P 0C ciuded exceeds the threshold, then the system 
classifies the image element as belonging to the foreground layer (140). 
Otherwise, the system classifies the image element as belonging to the 
background layer (130). An exemplary threshold lis 80%. The system then 
codes the image element using either a foreground or background coding 
scheme based on the element classification (150). 

[0025] Optionally, the system assigns some predetermined types of image 
elements to either the foreground or background layer. For example, the 
system may pre-assign text and symbols to the foreground layer. Since the 
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foreground may be encoded with higher resolution, encoding text and symbols 
as foreground improves their readability. 

[0026] It is to be understood that all or portions of the present invention 
may be used in connection with other methods for classifying image elements, 
as some aspects may be optional. For example, if text may be automatically 
assigned to the foreground layer, the method need not be applied to textual 
image elements. 

[0027] An image element processed according to the present invention 
may be initially represented as structured page information which describes the 
components of the image element and its coordinates in an image. Most of 
these components are simple operations such as "fill a rectangle," "draw a 
line," or "draw a piece of text." These operations simply assign a solid color to a 
set of specified portions called the element shape. Later operations may 
overwrite image data generated by earlier operations. When the printing 
operations are completed, they render a complete printed output (often a page). 

[0028] In the present invention, the system processes the image element 
from its structured page information, such that the image element coordinates 
are read and compared to foreground and background layers. From this 
comparison, the system determines the occluded portions of the image 
element. Then, the system renders the image element into either the 
foreground or background layers. 

[0029] Figure 2 is a flowchart of an exemplary method for calculating the 
image element perimeter. The method makes a number of Boolean operations 
between the foreground and background layers and the image element shapes. 
In this embodiment, these layers and shapes may be represented using run- 
length encoding. In run-length encoding, each scan line (or row) of the layers 
and image elements is represented by a sequence of lengths that describe 
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successive runs of black and white bits. A run is a contiguous group of 0's 
(white bits) or 1 's (black bits) encountered in a left to right scan of the scan line. 
For exemplary purposes, the black bits represent image elements and the white 
bits represent non-element space. Accordingly, the system calculates the 
perimeter of a run-length-encoded layer or element shape by making a single 
pass on the "black bit" runs, scanning each scan line of the layer or element 
shape from left to right (200). 

[0030] For each "black bit" run, the system calculates the run's perimeter r 
by adding twice the length of the run, i.e., the number of bits, and twice the 
width (205). The values of the length and width are doubled to account for, 
respectively, the top and bottom lengths and the end widths. Then, the system 
calculates the length / of each contact segment between the run and the 
adjacent run in the next scan line, i.e., the run bits in the next scan line that are 
adjacent to the run and are part of the same layer or element shape (210). 

[0031] After processing all the runs, the system sums all the perimeters rto 
produce R and all the contact lengths / to produce L (220). Since perimeters of 
adjacent runs include the same contact length, the system multiplies the sum L 
by two to account for the duplicate inclusion. The system calculates the 
perimeter of the layer or element shape as P= R-2L (230). 

[0032] Figure 3 illustrates the perimeter calculation in Figure 2 for an 
exemplary image element. In this example, the image element 300 is made up 
of 5 scan lines. Each scan line has a single "black bit" run, with the exception of 
the third scan line, which has two, separated by a "white bit" run. The system 
determines the perimeter rof each "black bit" run 310. Then, the system 
computes the sum Rof the perimeters rof the runs 310. The system detects 
the contact segments 320 between adjacent runs, sums all their lengths / , and 
multiplies the sum by 2 to produce 2L The perimeter P of the image element 
300 equals R-2L 
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[0033] It is to be understood that the perimeter calculation is for exemplary 
purposes only, as the perimeter calculated by other techniques may also be 
used by the present invention. Such techniques include, but are not limited to, 
contour-mapping and region-growing. 

[0034] Figure 4 illustrates an exemplary method for detecting the 
unoccluded and occluded portions of the image elements for which the system 
calculates perimeters using the method of Figure 2, for example, such that the 
image element may be classified as belonging to either foreground or 
background layers. First, the system creates two empty layers Fand Bto 
represent the image elements to be classified as foreground and background. 
The system then performs the following on every image element starting from 
the topmost element and proceeding toward the bottommost element. 

[0035] The system calculates the perimeter P ori g ina i of the image element 
(400). This is the perimeter of the original shape of the image element as it 
would appear without occlusion. Then, the system determines the portions of 
the image element shape occluded by background image elements drawn 
above the current image element (405). This is achieved by calculating the 
intersection of the image element shape and the current background B. For 
example, for a given image element portion i, if the current background portion 
B(i) has a value, then the image element portion / is designated as occluded 
and removed. 

[0036] For the first image element processed, i.e. the topmost element, the 
background B is empty, such that there are no occluded portions to remove. 
For subsequent elements, if the background includes previously processed 
occluding elements, the system removes the occluded portions from the image 
element shape (410). 
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[0037] Next, the system determines the portions of the resulting element 
shape occluded by foreground image elements drawn above the current image 
element (420). This is achieved by calculating the intersection of the image 
element shape and the current foreground F. For example, for a given image 
element portion /, if the current foreground portion F(i) has a value, then the 
image element portion /'is designated as occluded and removed. 

[0038] For the first image element processed, i.e., the topmost element, 
the foreground Fis empty, such that there are no occluded portions to remove. 
For subsequent elements, if the foreground includes previously processed 
occluding elements, the system removes the occluded portions from the image 
element shape (425). The system then calculates the perimeter Pension of the 
portions occluded by foreground and background elements (430). 

[0039] The resultant image element shape includes only the visible 
portions of the image element. The system now calculates the occluded 
perimeter Preluded of the element and the unoccluded perimeter Precluded of the 
element (440), where 

^occluded = R ~ 2/. , (2) 

the perimeter of the visible portions, as described previously. And, 

P + P - P 

p _ original occluded ' occlusion 

' unoccluded ^ ' \y) 

the perimeter of the visible boundary, where the sum of the occluded and 
original perimeters is equal to twice the unoccluded perimeter of the image 
element plus the perimeter of the occluded portions that are removed. 

[0040] Figures 5A-5C illustrate exemplary occluded and unoccluded 
portions of an image element detected according to a method of the present 
invention. Figure 5A illustrates a polygonal image element 700 which is 
occluded by *ab' image element 710. The occluded perimeter P 0C ciuded of the 
polygon 700, i.e., the perimeter of the visible portions with the occluded portions 
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removed, is shown in Figure 5B. The unoccluded perimeter P U nocciuded of the 
polygon 700, i.e., the discontinuous visible boundaries (indicated by thick black 
lines), is shown in Figure 5C. 

[0041] Figure 6 illustrates the calculation from Equation (3) used to 
determine the unoccluded perimeter of the polygon 700. The sum of the 
original and occluded perimeters, P orig inai and P 0C ciuded, is equal to twice the 
unoccluded perimeter P U nocciuded plus the perimeter of the occluded portions 

Pocclusion- 

[0042] It is to be understood that the method for detecting unoccluded and 
occluded portions of the image elements is for exemplary purposes only, as 
many variations of this method may be used by the present invention. 

[0043] Optionally, a color difference S may be included in the ratio, 

8P 

unoccluded ^ y- 
^occluded 

The color difference may be used to determine how closely the color of an 
image element matches that of the background. If the colors are closely 
matched, then the edges of the image element are not very distinct, such that 
the likelihood of the element belonging to the background is high. 

[0044] Generally, multiple passes over an image element are used to 
determine whether the element belongs in the foreground or background. This 
is done because elements located below the current element may turn out to be 
foreground elements, rather than background elements, and may thus affect the 
perimeter of the current element. In the present invention, the calculation of the 
color difference allows an estimate of the likelihood of the current element being 
assigned to background, thus enabling a single pass over an image element to 
determine whether it should be foreground or background. 
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[0045] Since the computation of the color difference may be very 
expensive, two exemplary simplifications may be used. Figure 7A is a flowchart 
of a first example. The system calculates the color difference between the 
element boundary and each of the adjacent background elements (500). Then, 
the system selects the largest of these color differences as S (510). 

[0046] Figure 7B illustrates a second example. The system selects a 
predetermined number of background elements (550). The system heuristically 
determines these background elements during a preliminary pass over the 
image elements according to the background elements' overall size and form. 
That is, an image element, such as a solid rectangle, with a large area 
compared to its bounding box (i.e., an imaginary boundary around the element) 
and a large area overlapping with the current image element is a potential 
background element. As such, the image element may be used to calculate the 
color difference. Conversely, an image element, such as a long thin curve, with 
a small area compared to its bounding box and a small area overlapping with 
the current image element is not a potential background element. As such, the 
image element may not be used to calculate the color difference. 

[0047] After the system selects the background elements, the system 
calculates the color difference between each of the selected adjacent 
background elements and the element boundary (560). The system then 
averages the color differences as S (570). 

[0048] It is to be understood that the methods described for calculating 
color difference are for exemplary purposes only, as many techniques to 
calculate color difference may be used in the present invention. Such 
techniques include, but are not limited to, gradient measurements. 

[0049] Figure 8 is a flowchart of a method according to another 
embodiment of the present invention. According to the invention, an image 
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classification system initializes a foreground layer Fand a background layer B 
to empty (580). Then, the system iteratively classifies each image element 
(582). 

[0050] For each image element, the system performs the following. The 
system determines the portions of the image element that intersect with the 
foreground and background layers (583), then identifies the intersecting 
portions as occluded portions of the image element and removes them (584). 
The system then calculates an unoccluded perimeter of the image element, 
Punocciuded, (585) and an occluded perimeter of the image element, Pocciuded 
(586). The system compares a ratio of these calculated perimeters to a 
predetermined threshold 7(587), as in Equation (1). If the ratio of Precluded to 
Pocciuded exceeds the threshold, then the system classifies the image element as 
belonging to the foreground layer (589). Otherwise, the system classifies the 
image element as belonging to the background layer (588). An exemplary 
threshold T is 80%. Optionally, the system includes a color difference in the 
ratio, as in Equation (4). 

[0051 ] After all the image elements are classified, the system then codes 
the foreground and background layers using their respective coding schemes 
(590). 

[0052] Figure 9 shows an exemplary electronically produced image, which 
may be segmented according to the present invention. The system 
decomposes the image 600 into a sequence of elementary image elements 610 
through 655 drawn on top of each other, where the front 610 and the back 655 
of the sequence indicate the image elements that are topmost and bottommost 
in the sequence, respectively. That is, the system prints the front image 
element last and the back image element first. The system may print the 
elementary image elements 610 through 655 separately as illustrated in Figure 
9 using printing software that processes the given image format. 
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[0053] According to the present invention, the system first prepares empty 
foreground and background layers Fand B representing the image elements to 
be classified as foreground and background. Then, the system processes the 
first element 610 according to the method of the present invention. That is, the 
system calculates the perimeter P or iginai of the original shape of the first element 
using the perimeter calculation of Figure 2, for example. Then, the system 
determines the portions of the first element shape that are occluded by 
background and foreground image elements drawn above the current image 
element as in Figure 4, for example. Since the layers Fand B are empty at this 
point, none of the first element is occluded and none of its portions are 
removed. So, the perimeter P OC ciusion of these non-existent occluded portions is 
nil. 

[0054] Now, the system computes the perimeters P 0C ciuded and P un occiuded of 
the first element from Equations (2) and (3). Optionally, the system may 
calculate the color difference between the first element and the background 
according to either method of Figures 7A or 7B, for example. Then, the system 
calculates the ratio of the occluded and unoccluded perimeters. Based on the 
ratio, the system classifies the first element as belonging to either the 
foreground or background layer, as shown in Figure 1 B, for example. Since the 
first element is unoccluded, the occluded and unoccluded perimeters are the 
same. Thus, the first element has a ratio of 1 or, with the color difference, 
higher. If the threshold is set at 80% (i.e., 0.8), then the first element is 
assigned to the foreground. The system then updates the foreground layer Fto 
include the first element. 

[0055] In the case of an occluded image element 650, which is occluded 
by elements 610, 615, 620, 630, and 635, the layers Fand Bmay not be empty 
because elements 610 through 645 will have been processed and classified 
prior to element 650. Thus, the system determines and removes the occluded 
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parts of the element 650. The occluded perimeter Preluded of the element 650 
may be larger than the unoccluded perimeter Precluded- As such, the ratio 
between the perimeters may be lower than the threshold. In which case, the 
system classifies the element 650 as background. 

[0056] The system repeats the method of the present invention for the 
image elements 610 through 655, resulting in classified image elements, which 
the system then efficiently encodes. 

[0057] The mechanisms and methods of the present invention may be 
implemented using a general-purpose microprocessor programmed according 
to the teachings of the present invention. The present invention thus also 
includes a machine-readable medium which includes instructions which can be 
executed by a processor to perform a method in accordance with the present 
invention. This medium can include, but is not limited to, any type of disk 
including floppy disk/optical disk, CD-ROMs, or any type of media suitable for 
storing electronic instructions. 

[0058] Figure 1 0 is a block diagram of one embodiment of a computer 
system that can implement the present invention. The system 700 may include, 
but is not limited to, a bus 710 in communication with a processor 720, a 
system memory module 730, and a storage device 740 according to 
embodiments of the present invention. 

[0059] It is to be understood that the structure of the software used to 
implement the invention may take any desired form. For example, the method 
illustrated may be implemented in a single program or multiple programs. 
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[0060] Numerous modifications and variations of the present invention are 
possible in light of the above teachings. It is therefore to be understood that 
within the scope of the appended claims, the invention may be practiced 
otherwise than as specifically described herein. 
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